Method, device, and computer-readable medium for mobile device management of collaborative industrial robot

ABSTRACT

Provided is a method, device, and computer-readable medium for controlling a robot graphic user interface (“RGUI”) on a mobile device. The method can include determining a distance, a position, or both of the mobile device with respect to a first robot; and causing, by a processor, a first RGUI to be displayed on a display of the mobile device based on the determining.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/854,779 filed on Apr. 21, 2020, which is a continuation of U.S.patent application Ser. No. 15/341,136 filed on Nov. 2, 2016, now U.S.Pat. No. 10,657,802 issued on May 19, 2020, which claims priority toU.S. Provisional Patent Application 62/249,686 filed on Nov. 2, 2015,all of which are incorporated by reference in their entirety.

GOVERNMENT SUPPORT STATEMENT

This invention was made with Government support under Grant No.NRI-1227277 awarded by the National Science Foundation. The U.S.Government has certain rights in this invention.

FIELD

The present disclosure relates generally to systems and methods forenabling human-machine collaborations via a generalized framework thatsupports dynamic adaptation and reuse of robotic capabilityrepresentations and human-machine collaborative behaviors.

BACKGROUND

Robotic industrial automation has seen significant success inlarge-scale manufacturing because it offers significant advantages atscale for tasks such as welding, cutting, stamping, painting, heavymaterial handling, precision material machining, etc. The success ofrobotic automation in large-scale manufacturing has led to along-standing desire to extend the use of robotic automation into smalland medium-sized manufacturing enterprises (“SMEs”). However, incontrast to large scale manufacturing, SMEs' production processes aretypically characterized by small production volumes and/or high productvariability. Consequently, the ability to amortize the infrastructure,specialized personnel, setup, and programming of flexible roboticautomation is far reduced for SMEs.

SME processes sometimes include tasks that require a high level ofcustomization and therefore necessarily involve human skill andjudgment. For example, refurbishment tasks and build-to-ordermanufacturing processes must accommodate unforeseen workpiece variancesand equipment modifications. In such cases, an existing human-centeredproduction process may find it difficult to determine where or howrobotic automation can be a useful addition to an effectivehuman-intensive process, rather than a duplication or attenuationthereof. Take, for instance, an SME specializing in custom furnituremanufacturing that has a number of highly-skilled employees. That SMEmay want to improve the efficiency and productivity of its employees byusing robotic systems to automate repetitive tasks that involvedexterous actions, such as drilling or sanding tasks. However, acommercial off-the-shelf robotic system would not be useful in this casebecause it would be impossible for the SME to leverage its employees'existing task knowledge and experience.

The teach pendant is a mainstay of industrial robots. Usually a controlpanel or touch screen attached by tether to the robot, the teach pendantis used for controlling all aspects of the robot, from writing code toprogramming actions. However, because the teach pendant is connected tothe robot using wires, the programmer needs to be in close proximity ofthe robot in order to use the teach pendant. This close proximity maynot be wanted or even desired due to safety considerations. Also, havinga different teach pendant for each robot may make robot control andmanagement a difficult and time consuming process.

There is therefore a need for systems and methods for overcoming theseand other problems presented by the prior art.

SUMMARY

In accordance with examples of the present disclosure, a method ofcontrolling a robot graphic user interface (“RGUI”) on a mobile deviceis provided. The method comprises determining a distance, a position, orboth of the mobile device with respect to a first robot; and causing, bya processor, a first RGUI to be displayed on a display of the mobiledevice based on the determining.

In some examples, the method further comprises determining a distance, aposition, or both of the mobile device with respect to a second robot;and causing, by a processor, a second RGUI to be displayed on thedisplay of the mobile device based on the determining

In some examples, the distance, the position, or both is determinedbased on one or more wireless protocols. In some examples, the one ormore wireless protocols comprise one or more of: WiFi, Bluetooth, RFID,cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC.

In some examples, the determining comprises assigning a RGUI for themobile device based on a waypoint on a user, the robot, or both; andselecting the RGUI based on the waypoint.

In some examples, the first RGUI changes as the distance, the position,or both of the mobile device changes.

In accordance with examples of the present disclosure, a device ofcontrolling a robot graphic user interface (“RGUI”) is provided. Thedevice comprises a memory containing instructions; and at least oneprocessor, operably connected to the memory, that executes theinstructions to perform operations comprising: determining a distance, aposition, or both of the device with respect to a first robot; andcausing, by a processor, a first RGUI to be displayed on a display ofthe device based on the determining

In some examples, the at least one processor is further operable toperform the method comprising: determining a distance, a position, orboth of the device with respect to a second robot; and causing, by aprocessor, a second RGUI to be displayed on the display of the devicebased on the determining; and causing by a processor, a state change onthe robot based on the determining. In some examples, the distance, theposition, or both is determined based on one or more wireless protocols.In some examples, the one or more wireless protocols comprise one ormore of: WiFi, Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee, Z-Wave,NFC. In some examples, the determining comprises: assigning a RGUI forthe device based on a docking location on a user, the robot, or both;and selecting the RGUI based on the docking location; and causing astate change on the robot based on the docking location. In someexamples, the first RGUI changes as the distance, the position, or bothof the device changes.

In accordance with examples of the present disclosure, acomputer-readable medium computer-interpretable instructions which, whenexecuted by at least one electronic processor, cause the at least oneelectronic processor to perform a method of controlling a robot graphicuser interface (“RGUI”) on a mobile device is provided. The methodcomprises determining a distance, a position, or both of the mobiledevice with respect to a first robot; and causing, by a processor, afirst RGUI to be displayed on a display of the mobile device based onthe determining. In some examples, the computer-readable medium furthercomprising determining a distance, a position, or both of the mobiledevice with respect to a second robot; and causing, by a processor, asecond RGUI to be displayed on the display of the mobile device based onthe determining; and causing, by a processor, a state change to occur onthe robot. In some examples, the distance, the position, or both isdetermined based on one or more wireless protocols. In some examples,the one or more wireless protocols comprise one or more of: WiFi,Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC. In someexamples, the determining comprises: assigning a RGUI for the mobiledevice based on a docking location on a user, the robot, or both; andselecting the RGUI based on the docking location; and triggering a statechange on the robot. In some examples, the first RGUI changes as thedistance, the position, or both of the mobile device changes.

In accordance with the present disclosure, a method for controlling arobot graphic user interface (“RGUI”) on a mobile device is disclosed.The method comprises determining a first distance, a first position, orboth of the mobile device with respect to a first robot; determining afirst communication protocol between the mobile device and the firstrobot; and causing, by a processor, a first RGUI to be displayed on adisplay of the mobile device based on the first communication protocolthat is determined, wherein the first RGUI changes as one or more of thefirst distance, the first position, the first communication protocol ofthe mobile device changes.

Various additional features can be included in the method including oneor more of the following features. The method further comprisesdetermining a second distance, a second position, or both of the mobiledevice with respect to a second robot; determining a secondcommunication protocol between the mobile device and the second robot;and causing, by a processor, a second RGUI to be displayed on thedisplay of the mobile device based on the second communication protocolthat is determined. The first distance, the first position, or both isdetermined based on one or more wireless protocols. The one or morewireless protocols comprise one or more of: WiFi, Bluetooth, RFID,cellular, ANT+, IrDA, mobile device based on a waypoint on a user, thefirst robot, or both and selecting the different first RGUI based on thewaypoint. The first RGUI changes as the distance, the position, or bothof the mobile device changes. The first distance, the first position, orboth is determined based on one or more wired protocols.

In accordance with examples of the present disclosure, a device forcontrolling a robot graphic user interface (“RGUI”) is disclosed. Thedevice comprises a memory containing instructions; and at least oneprocessor, operably connected to the memory, that executes theinstructions to perform operations comprising: determining a firstdistance, a first position, or both of the device with respect to afirst robot; determining a first communication protocol between thedevice and the first robot; and causing, by a processor, a first RGUI tobe displayed on a display of the device based on the first communicationprotocol that is determined, wherein the first RGUI changes as one ormore of the first distance, the first position, the first communicationprotocol of the device changes.

Various additional features can be included in the method including oneor more of the following features. The at least one processor is furtheroperable to perform the method comprising: determining a seconddistance, a second position, or both of the device with respect to asecond robot; determining a second communication protocol between thedevice and the second robot; and causing, by a processor, a second RGUIto be displayed on the display of the device based on the secondcommunication protocol that is determined. The first distance, the firstposition, or both is determined based on one or more wireless protocols.The one or more wireless protocols comprise one or more of: WiFi,Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC. Thedetermining comprises assigning different first RGUI for the devicebased on a waypoint on a user, the first robot, or both; and selectingthe different first RGUI based on the waypoint. The first RGUI changesas the distance, the position, or both of the device changes. The firstdistance, the first position, or both is determined based on one or morewired protocols.

In accordance with examples of the present disclosure, acomputer-readable medium comprising computer-interpretable instructionswhich, when executed by at least one electronic processor, cause the atleast one electronic processor to perform a method for controlling arobot graphic user interface (“RGUI”) on a mobile device, the methodcomprising: determining a first distance, a first position, or both ofthe mobile device with respect to a first robot; determining a firstcommunication protocol between the mobile device and the first robot;and causing, by a processor, a first RGUI to be displayed on a displayof the mobile device based on the first communication protocol that isdetermined, wherein the first RGUI changes as one or more of the firstdistance, the first position, the first communication protocol of themobile device changes.

Various additional features can be included in the method including oneor more of the following features. The computer-readable medium furthercomprises determining a second distance, a second position, or both ofthe mobile device with respect to a second robot; determining a secondcommunication protocol between the mobile device and the second robot;and causing, by a processor, a second RGUI to be displayed on thedisplay of the mobile device based on the second communication protocolthat is determined. The first distance, the first position, or both isdetermined based on one or more wireless protocols. The one or morewireless protocols comprise one or more of: WiFi, Bluetooth, RFID,cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC. The determining comprises:assigning different first RGUI for the mobile device based on a waypointon a user, the first robot, or both; and selecting the different firstRGUI based on the waypoint. The first RGUI changes as the distance, theposition, or both of the mobile device changes.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are not necessarily to scale. Instead, emphasis isgenerally placed upon illustrating the principles of the disclosuresdescribed herein. The accompanying drawings, which are incorporated inand constitute a part of this specification, illustrate severalembodiments consistent with the disclosures and together with thedescription, serve to explain the principles of the disclosures. In thedrawings:

FIG. 1 shows an example scenario where a mobile device is docked with arobot, where the RGUI provides for an interface that allows forinteraction of movement control of the robot, according to embodiments.

FIG. 2 shows an example scenario where mobile device is mounted on auser's forearm, where RGUI provides for an interface that allows forsetting of one or more waypoints for robot, according to embodiments.

FIG. 3 shows example different usage scenarios for RGUI based ondistance from a robot, according to embodiments.

FIG. 4 shows an example system diagram for a mobile device having a RGUIinteracting with two robots, according to embodiments.

FIG. 5 shows example usage scenario for a mobile device having differentRGUIs based on position on a user or a robot, according to embodiments.

FIG. 6 shows an example usage scenario for a mobile device having a RGUIfor two robots, according to embodiments

FIG. 7 shows an example computer system according to embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.Also, similarly-named elements perform similar functions and aresimilarly designed, unless specified otherwise. Numerous details are setforth to provide an understanding of the embodiments described herein.The embodiments may be practiced without these details. In otherinstances, well-known methods, procedures, and components have not beendescribed in detail to avoid obscuring the embodiments described. Whileseveral exemplary embodiments and features are described herein,modifications, adaptations, and other implementations are possible,without departing from the spirit and scope of the disclosure.Accordingly, the following detailed description does not limit thedisclosure. Instead, the proper scope of the disclosure is defined bythe appended claims.

In the present disclosure, the word “robot” will be used instead ofrobotic manipulator or set of robotic manipulators. Typically, a robotis an industrial robotic manipulator or a set of industrial roboticmanipulators for automated or semi-automated production applications. Arobot's envelope space is the range of motion over which the robot canphysically move or reach, which includes a set of points in space thatcan be reached by the robot's end-effector, which can be a tool effectorattached to the robot or a tool grasped or held by a gripper-typeend-effector attached to the robot. For example, the robot's envelopespace can include the range of motion over which a tool effector point(“TEP”) of the robot can physically move or reach. The robot's TEP canbe defined as a reference point in a point cloud of the tool effectorattached to the robot (e.g., a user-selected point, the tool effector'sendpoint, its point of attachment to the robot, etc.) or the toolgrasped by the robot's end-effector (e.g., a user-selected point, thetool's endpoint, the point at which the end-effector grasps the tool,etc.). The size and shape of the robot's envelope space depend on thecoordinate geometry of the robot and are influenced by the robot'sdesign, such as the robot's configuration (e.g., types of joints, thejoints' range of movement, lengths of links connecting the joints,etc.), number of degrees of freedom (“DOF”), and the like. In someembodiments, the size and shape of the robot's envelope space can alsobe influenced by the size and shape of the tool effector attached to therobot or the tool grasped by the robot's end-effector. The robot canperform work within its maximum workspace, which generally contains allor almost all of the points in the robot's envelope space. Thecollaborative system can constrain the robot's workspace to a subset ofthe points in the robot's maximum workspace to enforce or apply one ormore constraints, which is described in greater detail below.

In accordance with the various aspects of the present disclosure, asystem and method are provided whereby a mobile device (phone, tablet,wireless device) can be used in lieu of this teach pendant. This allowsfor greater mobility of the robot programmer at the site, allows foreasy access to commonly used robot commands, and also allows for changesin human-robot interactive behavior based on distance between the userand robot. This also conforms better to the increasingly prevalentmethod of physically interacting with human safe robots, where the useris physically guiding the robot. A mobile device allows the user to haverobot controls at-the-ready while guiding the robot.

In general the present disclosure can have the following features. Arobot graphical user interface (RGUI), which is present on a mobiledevice with a touch screen. The RGUI can accept touch, voice, gestural,acceleration-based or stylus-based input, and can display visual,vibration and auditory output. The mobile device could also allow fordisplay via an immersive virtual or augmented reality display. Themobile device can be held in hand during robot interaction, can beattached to the body of the user via an armband or sleeve, or can bedocked to various positions on the robot. Based on the position of thedevice, the RGUI can change modes.

The mobile device itself can recognize its position on the robot oruser, based on RFID tags placed at certain “docking locations” on therobot or user. Such locations include the end effector of the robot, thebase of the robot, an armband on the user, in the user's pocket, or in acustom cradle for holding the wireless device, that is not located on ornearby a robot. The mobile device can also recognize its position, or achange in location, based on camera sensing, inertial sensing or throughspecific interaction via the user (for instance, the user presses abutton titled “DOCK TO ROBOT”. Additionally, physical changes in thelocation or docking position of the mobile device can trigger changes ofmode on the robot.

Based on the distance from one or more robots, the RGUI can changemodes. For instance, assuming there are two robots in the facility, whenthe user is more than 5 meters away from both robots, the RGUI shows anoverview of what the two robots are currently working on. When the userapproaches one robot, closer than 5 meters, but further that 1 meter,the RGUI shows a more detailed overview for that particular robot, buthigh priority messages about the other robot can be reported. When theuser gets close to the robot to interact with it, the RGUI could show adetailed interface for a specific action, such as adding waypoints tothe robot's program, as described further below with reference toFIG. 1. A multitude of possible displays can be shown by the RGUI fordifferent distances, different docking positions, and/or number ofrobots. These can be selected as options by the user to be triggered bythe distance events.

The distance-based changes in the RGUI can also adapt to different modesthe robot is in. If the robot is moving and executing a task,approaching the robot might trigger a warning on the RGUI, orinformation about the specific action. Approaching the robot in teachingmode might trigger the RGUI to offer an interface for teaching therobot.

Based on the number of active or nearby robots, the RGUI can changemodes. Similarly, if several robots are operating simultaneously,updates on their progress and current state that are displayed in theRGUI can be prioritized based on which robot is closest to the user.

The system operates as follows. One or more robotic system is equippedwith a communication layer. This layer allows for wireless devices, or acentral server to communicate with the robot. A robotic graphical userinterface (RGUI) runs on the mobile device. This user interface canchange display and input modalities depending on which interactionmodality is being used. The RGUI can detect acceleration events, touchscreen presses, and voice input. These actions can trigger robot events,programming calls, processes or other events on the robotic system. Acommunication manager (CM) also runs on the mobile device. The CMcommunicates wirelessly with either the robots (via wifi or bluetooth,or other protocol) or a central server (via wifi or bluetooth, or otherprotocol). The CM also incorporates an RFID process that uses thewireless devices' onboard RFID reader.

The following distance metric can be used to switch between RGUIinteraction modalities: when the CM recognizes one or more robot withinshort range RFID; when the CM recognizes a robot is within short rangeRFID, and one or more other robot is within long range RFID; when the CMrecognizes one or more robot within long range RFID; when the CMrecognizes that no robots are within long range RFID, but robots arestill found on the network communication layer; when the CM recognizesvia short range RFID that the wireless device has been placed on aspecific part of the robot; when the CM recognizes via short range RFIDthat the wireless device has been placed in a docking position on theuser (for instance, on a cradle attached to a glove or arm band).

The following mounting options can be selected by the user to trigger achange in RGUI interaction modalities: the mobile device is placed on aforearm position of a user; the mobile device is docked to the base ofthe robot; the mobile device is docked to the end-effector region of therobot; the mobile device is placed in the pocket of the user.

Communication that takes place between the wireless device and the robotcan contain but is not limited to the following information: robot taskstate; robot internal state; warning/error messages; feedback onprogramming progress; task progress.

In some aspects, the robot can trigger changes to the RGUI based on itsstate, even though the device may have not changed location.

FIG. 1 shows an example scenario 100 where a mobile device is dockedwith a robot, where the RGUI provides for an interface that allows forinteraction of movement control of the robot, according to embodiments.As shown in FIG. 1, robot 105 has an end effector 110 in the form of agrasping member and mobile device dock 115 for providing an electricalconnection with mobile device 120. Mobile device 120 includes a RGUI 125that allows for interaction with robot 105. In the robot docketed stateof FIG. 1, RGUI 125 provides for movement monitoring and control of endeffector 110. RGUI 125 can include robot identity descriptor portion130, end effector control portion 135, for example “open” or “close,”end effector action description portion 140, an end effector movementindictor portion 145 that shows movement indicators for up, down, left,and right.

FIG. 2 shows an example scenario 200 where mobile device 120 is mountedon a user's forearm, where RGUI 125 provides for an interface thatallows for setting of one or more waypoints for robot 105, according toembodiments.

FIG. 3 shows example different usage scenarios 300 for RGUI based ondistance from a robot, according to embodiments. As shown in FIG. 3,mobile device 120 can be operable to display a different RGUI dependingon a distance from robot 105. For short range distances 320, such aswithin a RFID region, RGUI A 305 can be displayed on mobile device 120.For mid-range or long range distances 325, RGUI B 310 can be displayedon mobile device 120. For longer range distances or out of RFID rangedistances 330, RGUI C 315 can be displayed on mobile device 120. Thedistance can be determined based on one or more technologies and/orprotocols, such as WiFi, Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee,Z-Wave, NFC, etc. For example, one or more RFID tags can be positionedon robot 105 and mobile device can be operable to determine a distancefrom the one or more RFID tags.

FIG. 4 shows an example system diagram 400 for a mobile device having aRGUI interacting with two robots, according to embodiments. Mobiledevice 120 includes, among other feature, RGUI 410 and communicationmanger 325. Communication manager 325 can wirelessly communicate withrobot A 305, robot B 310, and server 405. RGUI 410 can change dependingon a distance from of mobile device 120 to robot A and/or robot B or theposition of mobile device 120 with respect to robot A 305, robot B 310,and the user.

FIG. 5 shows example usage scenario 500 for a mobile device havingdifferent RGUIs based on position on a user or a robot, according toembodiments. As shown in FIG. 5, the RGUI on mobile device 120 canchange depending on position on user 505 and on robot 540. For example,mobile device 120 may be operable to display RGUI A 510 when in handposition 515, RGUI B 520 when in pocket position 525, RGUI C 530 when inforearm position 535, RGUI D 545 when at end effector docked position550, and RGUI E 555 when at robot base docked position 560.

FIG. 6 shows an example usage scenario 600 for a mobile device having aRGUI for two robots, according to embodiments. When mobile device 120 isin proximity of more than one robot (two are shown in this example),RGUI 605 can display information on each robot. As shown in FIG. 6, RGUI605 can include fields that show the number of robots in operation 615,robot A status 620 for robot A 650, activity parameters 625 for robot A650, robot B status 630, activity parameters 635 for robot B, and image640 of the closest robot, which in this example is robot A 650.

The foregoing description is illustrative, and variations inconfiguration and implementation can occur to persons skilled in theart. For instance, the various illustrative logics, logical blocks,modules, and circuits described in connection with the embodimentsdisclosed herein can be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor canbe a microprocessor, but, in the alternative, the processor can be anyconventional processor, controller, microcontroller, or state machine. Aprocessor can also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

In one or more exemplary embodiments, the functions described can beimplemented in hardware, software, firmware, or any combination thereof.For a software implementation, the techniques described herein can beimplemented with modules (e.g., procedures, functions, subprograms,programs, routines, subroutines, modules, software packages, classes,and so on) that perform the functions described herein. A module can becoupled to another module or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, or the like can be passed,forwarded, or transmitted using any suitable means including memorysharing, message passing, token passing, network transmission, and thelike. The software codes can be stored in memory units and executed byprocessors. The memory unit can be implemented within the processor orexternal to the processor, in which case it can be communicativelycoupled to the processor via various means as is known in the art.

For example, FIG. 7 illustrates an example of a hardware configurationfor a computer device 700 that can be used as mobile device 120, thatcan be used to perform one or more of the processes described above.While FIG. 7 illustrates various components contained in the computerdevice 700, FIG. 7 illustrates one example of a computer device andadditional components can be added and existing components can beremoved.

The computer device 700 can be any type of computer devices, such asdesktops, laptops, servers, etc., or mobile devices, such as smarttelephones, tablet computers, cellular telephones, personal digitalassistants, etc. As illustrated in FIG. 7, the computer device 700 caninclude one or more processors 702 of varying core configurations andclock frequencies. The computer device 700 can also include one or morememory devices 704 that serve as a main memory during the operation ofthe computer device 700. For example, during operation, a copy of thesoftware that supports the DNS operations can be stored in the one ormore memory devices 704. The computer device 700 can also include one ormore peripheral interfaces 706, such as keyboards, mice, touchpads,computer screens, touchscreens, etc., for enabling human interactionwith and manipulation of the computer device 700.

The computer device 700 can also include one or more network interfaces708 for communicating via one or more networks, such as Ethernetadapters, wireless transceivers, or serial network components, forcommunicating over wired or wireless media using protocols. The computerdevice 700 can also include one or more storage device 710 of varyingphysical dimensions and storage capacities, such as flash drives, harddrives, random access memory, etc., for storing data, such as images,files, and program instructions for execution by the one or moreprocessors 702.

Additionally, the computer device 700 can include one or more softwareprograms 712 that enable the functionality described above. The one ormore software programs 712 can include instructions that cause the oneor more processors 702 to perform the processes described herein. Copiesof the one or more software programs 712 can be stored in the one ormore memory devices 704 and/or on in the one or more storage devices710. Likewise, the data, for example, DNS records, utilized by one ormore software programs 712 can be stored in the one or more memorydevices 704 and/or on in the one or more storage devices 710.

In implementations, the computer device 700 can communicate with otherdevices via a network 716. The other devices can be any types of devicesas described above. The network 716 can be any type of network, such asa local area network, a wide-area network, a virtual private network,the Internet, an intranet, an extranet, a public switched telephonenetwork, an infrared network, a wireless network, and any combinationthereof. The network 716 can support communications using any of avariety of commercially-available protocols, such as TCP/IP, UDP, OSI,FTP, UPnP, NFS, CIFS, AppleTalk, and the like. The network 716 can be,for example, a local area network, a wide-area network, a virtualprivate network, the Internet, an intranet, an extranet, a publicswitched telephone network, an infrared network, a wireless network, andany combination thereof.

The computer device 700 can include a variety of data stores and othermemory and storage media as discussed above. These can reside in avariety of locations, such as on a storage medium local to (and/orresident in) one or more of the computers or remote from any or all ofthe computers across the network. In some implementations, informationcan reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers, or other network devices may bestored locally and/or remotely, as appropriate.

In implementations, the components of the computer device 700 asdescribed above need not be enclosed within a single enclosure or evenlocated in close proximity to one another. Those skilled in the art willappreciate that the above-described componentry are examples only, asthe computer device 700 can include any type of hardware componentry,including any necessary accompanying firmware or software, forperforming the disclosed implementations. The computer device 700 canalso be implemented in part or in whole by electronic circuit componentsor processors, such as application-specific integrated circuits (ASICs)or field-programmable gate arrays (FPGAs).

If implemented in software, the functions can be stored on ortransmitted over a computer-readable medium as one or more instructionsor code. Computer-readable media includes both tangible, non-transitorycomputer storage media and communication media including any medium thatfacilitates transfer of a computer program from one place to another. Astorage media can be any available tangible, non-transitory media thatcan be accessed by a computer. By way of example, and not limitation,such tangible, non-transitory computer-readable media can comprise RAM,ROM, flash memory, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any othermedium that can be used to carry or store desired program code in theform of instructions or data structures and that can be accessed by acomputer. Disk and disc, as used herein, includes CD, laser disc,optical disc, DVD, floppy disk and Blu-ray disc where disks usuallyreproduce data magnetically, while discs reproduce data optically withlasers. Also, any connection is properly termed a computer-readablemedium. For example, if the software is transmitted from a website,server, or other remote source using a coaxial cable, fiber optic cable,twisted pair, digital subscriber line (DSL), or wireless technologiessuch as infrared, radio, and microwave, then the coaxial cable, fiberoptic cable, twisted pair, DSL, or wireless technologies such asinfrared, radio, and microwave are included in the definition of medium.Combinations of the above should also be included within the scope ofcomputer-readable media.

While the teachings have been described with reference to examples ofthe implementations thereof, those skilled in the art will be able tomake various modifications to the described implementations withoutdeparting from the true spirit and scope. The terms and descriptionsused herein are set forth by way of illustration only and are not meantas limitations. In particular, although the processes have beendescribed by examples, the stages of the processes can be performed in adifferent order than illustrated or simultaneously. Furthermore, to theextent that the terms “including”, “includes”, “having”, “has”, “with”,or variants thereof are used in the detailed description, such terms areintended to be inclusive in a manner similar to the term “comprising.”As used herein, the terms “one or more of” and “at least one of” withrespect to a listing of items such as, for example, A and B, means Aalone, B alone, or A and B. Further, unless specified otherwise, theterm “set” should be interpreted as “one or more.” Also, the term“couple” or “couples” is intended to mean either an indirect or directconnection. Thus, if a first device couples to a second device, thatconnection can be through a direct connection, or through an indirectconnection via other devices, components, and connections.

Those skilled in the art will be able to make various modifications tothe described embodiments without departing from the true spirit andscope. The terms and descriptions used herein are set forth by way ofillustration only and are not meant as limitations. In particular,although the method has been described by examples, the steps of themethod can be performed in a different order than illustrated orsimultaneously. Those skilled in the art will recognize that these andother variations are possible within the spirit and scope as defined inthe following claims and their equivalents.

The foregoing description of the disclosure, along with its associatedembodiments, has been presented for purposes of illustration only. It isnot exhaustive and does not limit the disclosure to the precise formdisclosed. Those skilled in the art will appreciate from the foregoingdescription that modifications and variations are possible in light ofthe above teachings or may be acquired from practicing the disclosure.For example, the steps described need not be performed in the samesequence discussed or with the same degree of separation. Likewisevarious steps may be omitted, repeated, or combined, as necessary, toachieve the same or similar objectives. Similarly, the systems describedneed not necessarily include all parts described in the embodiments, andmay also include other parts not describe in the embodiments.

Accordingly, the disclosure is not limited to the above-describedembodiments, but instead is defined by the appended claims in light oftheir full scope of equivalents.

1. A method for controlling a robot graphic user interface (“RGUI”) on a mobile device, the method comprising: determining a first distance, a first position, or both of the mobile device with respect to a first robot; determining a first communication protocol between the mobile device and the first robot; and causing, by a processor, a first RGUI to be displayed on a display of the mobile device based on the first communication protocol that is determined, wherein the first RGUI changes as one or more of the first distance, the first position, the first communication protocol of the mobile device changes.
 2. The method of claim 1, further comprising: determining a second distance, a second position, or both of the mobile device with respect to a second robot; determining a second communication protocol between the mobile device and the second robot; and causing, by a processor, a second RGUI to be displayed on the display of the mobile device based on the second communication protocol that is determined.
 3. The method of claim 1, wherein the first distance, the first position, or both is determined based on one or more wireless protocols.
 4. The method of claim 3, wherein the one or more wireless protocols comprise one or more of: WiFi, Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC.
 5. The method of claim 1, wherein the determining comprises: assigning different first RGUI for the mobile device based on a waypoint on a user, the first robot, or both; and selecting the different first RGUI based on the waypoint.
 6. The method of claim 1, wherein the first RGUI changes as the distance, the position, or both of the mobile device changes.
 7. The method of claim 1, the first distance, the first position, or both is determined based on one or more wired protocols.
 8. A device for controlling a robot graphic user interface (“RGUI”), the device comprising: a memory containing instructions; and at least one processor, operably connected to the memory, that executes the instructions to perform operations comprising: determining a first distance, a first position, or both of the device with respect to a first robot; determining a first communication protocol between the device and the first robot; and causing, by a processor, a first RGUI to be displayed on a display of the device based on the first communication protocol that is determined, wherein the first RGUI changes as one or more of the first distance, the first position, the first communication protocol of the device changes.
 9. The device of claim 8, wherein the at least one processor is further operable to perform the method comprising: determining a second distance, a second position, or both of the device with respect to a second robot; determining a second communication protocol between the device and the second robot; and causing, by a processor, a second RGUI to be displayed on the display of the device based on the second communication protocol that is determined.
 10. The device of claim 8, wherein the first distance, the first position, or both is determined based on one or more wireless protocols.
 11. The device of claim 10, wherein the one or more wireless protocols comprise one or more of: WiFi, Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC.
 12. The device of claim 8, wherein the determining comprises: assigning different first RGUI for the device based on a waypoint on a user, the first robot, or both; and selecting the different first RGUI based on the waypoint.
 13. The device of claim 8, wherein the first RGUI changes as the distance, the position, or both of the device changes.
 14. The device of claim 8, wherein the first distance, the first position, or both is determined based on one or more wired protocols.
 15. A computer-readable medium comprising computer-interpretable instructions which, when executed by at least one electronic processor, cause the at least one electronic processor to perform a method for controlling a robot graphic user interface (“RGUI”) on a mobile device, the method comprising: determining a first distance, a first position, or both of the mobile device with respect to a first robot; determining a first communication protocol between the mobile device and the first robot; and causing, by a processor, a first RGUI to be displayed on a display of the mobile device based on the first communication protocol that is determined, wherein the first RGUI changes as one or more of the first distance, the first position, the first communication protocol of the mobile device changes.
 16. The computer-readable medium of claim 15, further comprising: determining a second distance, a second position, or both of the mobile device with respect to a second robot; determining a second communication protocol between the mobile device and the second robot; and causing, by a processor, a second RGUI to be displayed on the display of the mobile device based on the second communication protocol that is determined.
 17. The computer-readable medium of claim 15, wherein the first distance, the first position, or both is determined based on one or more wireless protocols.
 18. The computer-readable medium of claim 17, wherein the one or more wireless protocols comprise one or more of: WiFi, Bluetooth, RFID, cellular, ANT+, IrDA, ZigBee, Z-Wave, NFC.
 19. The computer-readable medium of claim 15, wherein the determining comprises: assigning different first RGUI for the mobile device based on a waypoint on a user, the first robot, or both; and selecting the different first RGUI based on the waypoint.
 20. The computer-readable medium of claim 15, wherein the first RGUI changes as the distance, the position, or both of the mobile device changes. 